Fault Injection for the Masses

Many people believe that software fault injection is a complex process that requires investing thousands of dollars in tools to perform Here we look at the most basic requirements for performing fault injection and how for limited software analysis the instrumentation needed to collect the results can be inserted manually thus avoiding the need for advanced tools Getting theoretical ideas into practice is a key reason I avoided academia The key technology that I would like to see adopted by the masses is a family of software fault injection algorithms that can predict where to concentrate testing as well as how much testing to do whether systems can tolerate anomalies and remain in safe states and whether systems can be broken into via malicious events There are other lesser applications that I will ignore here for brevity From a novelty standpoint these algorithms were and still are unique amongst other methods of performing fault injection I concede that the algorithms are computational but the results can provide unequaled information about how bad things propagate through systems Because of that I ve considered the methods valuable to anyone responsible for software quality This spans the spectrum from one person Independent Software Vendors ISVs to the largest corporations Moving this technology from its early adopters which have been large corporations and US government organizations to the masses which is mainly comprised of the smaller ISVs is no small challenge Even with testimonials such as Based on the successful use of fault injection on prior and on going programs we are moving toward standardizing on this technique for all future software intensive safety critical systems Larry James Senior Scientist Engineering Technology Center Hughes Infor mation Systems Fault injection is the next wave in the rapidly advancing eld of software quality assurance Barry Preppernau Test Training Manager Microsoft This is a fundamental technology that has no equal The principles and speci c methodologies have been greatly bene cial in increasing the e ciency and e ec tiveness of our software testing especially in the area of high reliability high availability systems Mike Friedman Hughes Electronics getting the masses to try out fault injection requires a user to take resources from one V V pot and place them in another As a general rule of thumb people become uncomfortable when asked to move from something familiar to something unfamiliar To most people fault injection is still a great unknown During the remainder of this piece I d like to share my thoughts on how the small resource tight organizations can begin making fault injection a software reality in their processes In fact you can start seeing results within hours by following the simple steps I will now walk through Many people think that to perform fault injection that they must have an expensive automated tool Rubbish Early on I did fault injection by hand It was not until several years later that I had access to a tool that automated portions of the analysis Then I coded in the results collecting and injecting instrumentation manually This clearly limited how much fault injection could occur but it also had a side bene t It forced me to think about where I wanted the injectors placed in the code By thinking more about placement I needed less instrumentation making manual analysis more feasible And it also decreased computational costs Now I m not suggesting that this is a long term solution that voids the need for automation but it does allow you to try out fault injection without acquiring tools You may be wondering how I decided where to place the fault injection probes as well as how many places in the code I could realistically perform this manual process The most that I recall doing was around mainly because the process is so repetitive that I would start making dumb mistakes like forgetting to delete previous fault injection instrumentation before embedding the new instrumentation What I would do would be to search through the code for those places where speci c variables that I feared might have wrong information in them were assigned values By applying fault injection to those places my fears were either con rmed or dispelled As an example I almost always opted to place fault injectors right after input informa tion was read in I was naturally curious about how my program would behave after bad information was fed into it and fault injection provided a reasonable way to satisfy this curiosity Such curiosity is justi ed and is felt by most software developers They know that assuming that information originating from external sources will be correct is dangerous The question then becomes how dangerous To make this manual process a little easier for the reader to visualize let s walk through a couple of examples Let s assume you have a sequence of statements in your code like this